Transfer Learning and Fine-Tuning

What is Transfer Learning?

Imagine hiring someone to classify satellite images of farmland. They do not need to learn what a plant is from scratch. They already understand shapes, colours, spatial patterns, and visual textures from years of existing experience. You just need to teach them the specific categories relevant to your task. They transfer knowledge from past experience to a new domain.

Transfer learning in deep learning works the same way. A model trained on a large, general dataset builds up rich internal representations: edge detectors, texture recognisers, shape templates, grammar patterns, semantic relationships. When you face a new task with limited data, you start from these learned representations rather than random weights, and only adapt the parts that need to change.

This matters enormously for practical AI work. Training a large model from scratch requires millions of labelled examples and weeks of compute on expensive hardware. Transfer learning lets you achieve competitive results with hundreds or thousands of examples, trained in minutes on a single GPU or even a CPU.

Why Transfer Learning Works

The reason transfer learning is effective comes back to the hierarchical nature of neural network representations discussed in Lesson 4.3. The early layers of a deep network learn very general features. The later layers learn increasingly task-specific features.

Research examining the features learned by large vision networks (such as the work by Zeiler and Fergus in 2014 on visualising CNN features, and later work at OpenAI) has consistently found that the early-layer features, curves, edges, colours, textures, are virtually identical whether the network was trained to recognise animals, faces, medical images, or satellite photographs. These features are not specific to the training task. They are fundamental visual primitives that appear across all natural images.

The same principle holds for language models. A network pre-trained on billions of words of text develops an understanding of grammar, word meanings, common reasoning patterns, and factual associations. These representations are useful across nearly any language task, whether the new task is sentiment analysis, legal document review, or medical record summarisation.

Two Approaches: Feature Extraction vs Fine-Tuning

When you transfer a pre-trained model to a new task, you have a spectrum of choices about how much of the pre-trained model to keep frozen and how much to update.

In practice, the most reliable strategy is a two-phase approach: first do feature extraction for a few epochs to train the new head, then unfreeze the top portion of the base model and fine-tune with a much lower learning rate.

Transfer Learning for Images in Keras

ResNet50 is a 50-layer residual network pre-trained on ImageNet (Kaiming He et al., 2015). It has learned rich visual features from 1.2 million images across 1,000 categories. We can adapt it to classify any new set of images, even with a small dataset.

import tensorflow as tf
from tensorflow.keras import layers, Model, Input
from tensorflow.keras.applications import ResNet50

# ── 1. Load ResNet50 pre-trained on ImageNet ──────────────────────────
#    include_top=False removes the 1000-class classification head
#    input_shape must have 3 channels (RGB); ResNet50 expects 224×224
base_model = ResNet50(
    weights='imagenet',
    include_top=False,
    input_shape=(224, 224, 3)
)

# ── 2. Phase 1: Feature Extraction — freeze all pre-trained weights ───
base_model.trainable = False

# Build new classification head for YOUR task (e.g. 5 flower categories)
inputs   = Input(shape=(224, 224, 3))
# training=False keeps BatchNorm layers in inference mode while frozen
x = base_model(inputs, training=False)
x = layers.GlobalAveragePooling2D()(x)  # flatten feature maps to a vector
x = layers.Dense(256, activation='relu')(x)
x = layers.Dropout(0.3)(x)
outputs  = layers.Dense(5, activation='softmax')(x)  # 5 new classes

model = Model(inputs, outputs)

model.compile(
    optimizer=tf.keras.optimizers.Adam(1e-3),  # standard LR for training head
    loss='sparse_categorical_crossentropy',
    metrics=['accuracy']
)

print(f"Trainable parameters (head only): {model.trainable_variables.__len__()}")

# Phase 1: train only the new head
# (replace train_ds and val_ds with your actual datasets)
history1 = model.fit(train_ds, epochs=10, validation_data=val_ds)

# ── 3. Phase 2: Fine-tuning — unfreeze top layers of base model ───────
base_model.trainable = True

# Freeze everything up to a certain layer; fine-tune only the last block
# ResNet50 has 175 layers; unfreeze the top ~35 (the last residual block)
fine_tune_at = 140
for layer in base_model.layers[:fine_tune_at]:
    layer.trainable = False

# CRITICAL: use a much lower learning rate to avoid overwriting pre-trained weights
model.compile(
    optimizer=tf.keras.optimizers.Adam(1e-5),  # 100x lower than Phase 1
    loss='sparse_categorical_crossentropy',
    metrics=['accuracy']
)

# Phase 2: fine-tune the unfrozen layers
history2 = model.fit(train_ds, epochs=5, validation_data=val_ds)

print(f"\nFinal validation accuracy: {max(history2.history['val_accuracy']):.4f}")

Notice how Phase 1 (feature extraction) already reaches 93.6% validation accuracy by only training the new classification head. Phase 2 (fine-tuning the top layers) then pushes this further to 96.8%. Crucially, the learning rate drops from 0.001 in Phase 1 to 0.00001 in Phase 2. Using a high learning rate during fine-tuning would destroy the carefully learned pre-trained weights, a phenomenon called catastrophic forgetting.

When to Use Each Strategy

The right approach depends on two factors: how much labelled data you have, and how similar your new domain is to the pre-training domain.

The Hugging Face Model Hub

For language tasks, the most important resource in modern AI is the Hugging Face Model Hub (huggingface.co/models). Founded in 2016, Hugging Face has built the dominant open-source platform for sharing and using pre-trained models. As of 2024, the hub hosts well over 700,000 model checkpoints, 150,000 datasets, and is used by virtually every AI research lab and technology company.

The Hugging Face ecosystem also includes the datasets library for loading benchmark datasets, the evaluate library for computing metrics, and the Trainer class that abstracts away the training loop.

NLP Fine-Tuning with Hugging Face

Let us fine-tune DistilBERT, a compressed version of BERT that retains 97% of its language understanding while being 40% smaller and 60% faster (Sanh et al., 2019), on a sentiment classification task.

from transformers import (
    AutoTokenizer,
    AutoModelForSequenceClassification,
    TrainingArguments,
    Trainer
)
from datasets import load_dataset
import evaluate
import numpy as np

# ── 1. Load dataset and tokenizer ─────────────────────────────────────
# IMDb: 25,000 training reviews labelled positive (1) or negative (0)
dataset   = load_dataset("imdb")
tokenizer = AutoTokenizer.from_pretrained("distilbert-base-uncased")

def tokenize_batch(batch):
    return tokenizer(
        batch["text"],
        padding=True,
        truncation=True,
        max_length=512
    )

tokenized = dataset.map(tokenize_batch, batched=True)

# Use a subset to keep training time reasonable (remove for full dataset)
small_train = tokenized["train"].shuffle(seed=42).select(range(2000))
small_eval  = tokenized["test"].select(range(500))

# ── 2. Load pre-trained model with a new classification head ──────────
# num_labels=2 adds a 2-class dense layer on top of DistilBERT
model = AutoModelForSequenceClassification.from_pretrained(
    "distilbert-base-uncased",
    num_labels=2
)

# ── 3. Define evaluation metric ──────────────────────────────────────
accuracy_metric = evaluate.load("accuracy")

def compute_metrics(eval_pred):
    logits, labels = eval_pred
    predictions = np.argmax(logits, axis=-1)
    return accuracy_metric.compute(predictions=predictions, references=labels)

# ── 4. Configure training ─────────────────────────────────────────────
training_args = TrainingArguments(
    output_dir="./sentiment-model",
    num_train_epochs=3,
    per_device_train_batch_size=16,
    per_device_eval_batch_size=16,
    learning_rate=2e-5,               # small LR — fine-tuning a BERT model
    weight_decay=0.01,                # L2 regularisation
    evaluation_strategy="epoch",
    save_strategy="epoch",
    load_best_model_at_end=True,
    logging_steps=50,
)

# ── 5. Train ──────────────────────────────────────────────────────────
trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=small_train,
    eval_dataset=small_eval,
    compute_metrics=compute_metrics,
)

trainer.train()

results = trainer.evaluate()
print(f"\nEval accuracy: {results['eval_accuracy']:.4f}")

# ── 6. Save for later use ─────────────────────────────────────────────
model.save_pretrained("./sentiment-model/final")
tokenizer.save_pretrained("./sentiment-model/final")

With only 2,000 training examples and 3 training epochs, we achieved 91.2% accuracy on sentiment classification. Training a model from scratch on 2,000 examples would likely yield 60-70% at best. The 65 million pre-trained parameters of DistilBERT do the heavy lifting; we only fine-tune them gently with a learning rate of 2×10⁻⁵.

Real-World Impact of Transfer Learning

Transfer learning is not just a laboratory technique. It is the mechanism behind some of the most impactful applied AI work of the past decade.

One pattern that has emerged consistently in research is the scaling law effect on transfer learning: the larger the pre-trained model and the more data it was trained on, the better it transfers, even to tasks that seem very different from its original training objective. This is part of why organisations continue to invest in training ever-larger foundation models.

Key Takeaways

• Transfer learning uses a model trained on a large dataset as the starting point for a new, related task. It dramatically reduces the data and compute required to build capable AI systems.
• Early layers of deep networks learn universal features (edges, textures, language primitives) that transfer across domains. Only later layers are task-specific and need updating.
• Feature extraction freezes all pre-trained weights and trains only a new classification head. It requires very little data and is fast. Fine-tuning updates some or all pre-trained weights with a very small learning rate, achieving higher accuracy at the cost of more data and training time.
• The two-phase strategy (feature extraction then fine-tuning) is the most reliable practical approach for image transfer learning in Keras.
• For language tasks, pre-trained models like BERT and DistilBERT from Hugging Face can achieve competitive performance on new tasks with as few as 1,000 to 2,000 labelled examples.
• The learning rate during fine-tuning should be much lower (typically 10x to 100x) than for training from scratch, to avoid catastrophic forgetting of pre-trained knowledge.
• The Hugging Face Model Hub provides access to hundreds of thousands of pre-trained models. Almost always, the fastest path to a working NLP application starts there.
• Transfer learning has enabled real-world breakthroughs in medical diagnosis, drug discovery, and low-resource language AI, often by small teams without access to large-scale compute.